Automating Experimentation with Distributed Systems Using Generative Techniques

failure scenario. Such a model regards node failure and recovery as random accidents in the system execution environment. However, for a distributed system deployed for real use, the exceptions are frequently caused by the incorrect operation or the heavy load on the distributed system. We further used Weevil to conduct an experiment with the CFS service network under such realistic failure scenarios. In real setup of a distributed system, the system components with heavier load have higher probability to fail than the others. This experiment is to model such a fault injection scenario that is reactive to the real system execution status. Our experimental goal is to testify that the failure of the heavier-loaded CFS nodes influences more on the system performance than that of the randomly selected CFS nodes. We considered a 100-node CFS system distributed on a 50-host Emulab testbed with 2,000 files already inserted. Each CFS node had a user actor constantly issuing 50 retrieval requests for randomly chosen file keys. We reused the user workload generated for the previous continuous fault injection experiment. Since we cannot decide which CFS node to fail before trial execution, we associated an environment actor with each CFS node. Each actor’s portion of the environment workload is a list of conditional “STARTUP” and “BRINGDOWN” actions in a constant interval. Whether they would be issued or not depends on the two variables of the associated CFS node, the rpcrate and the liveornot. If the node’s remote pro-